Network-aware worker placement for wide-area streaming analytics

Many organizations leverage Distributed Stream processing systems (DPSs) to get insights from the data generated by different users/devices, e.g., Internet of Things (IoT) devices or user clicks on a website, geographically distributed datacenters. The worker nodes in such environments are connected through Wide Area Network (WAN) links with various delays and bandwidth. Therefore, minimizing execution latency task while using enough bandwidth lower cost steer traffic applications is challenging task. In this paper, we formulate node placement for geo-distributed DSPs network as multi-criteria decision-making problem. Then, propose an additive weighting-based approach solve it. users can prioritize according network-relevant parameters. We also framework that be integrated current DPSs execute tasks. test our three widely used stream systems, i.e., Apache Spark, Storm, Flink, custom graphs adopted real cloud providers. run streaming query Yahoo! benchmark these DPSs. experimental results show improves performance Spark up 2.2x–7.2x, Storm 1.2x–3.4x, Flink 1.4x–3.3x compared other approaches, which makes useful use practical environments.